Systems and methods for improved montaging of ophthalmic imaging data

ABSTRACT

An ophthalmic imaging system provides a user interface to facilitate the montaging of scan images collected with various imaging modalities, such as images collected with a fundus imaging system or an optical coherence tomography (OCT) system. The amount of each constituent image used in the montage is dependent upon its respective quality. During the collecting of scans (constituent images) for montaging, any scan may be designated for rescanning, such as if its current quality is deemed less than sufficient. In the case of using an OCT system to collect constituent images (e.g., cube scans), the scanned region of a constituent image may be modified based on physical characteristics of the eye being scanned.

PRIORITY

This application claims priority to U.S. Provisional Application Ser.No. 62/555,442 filed Sep. 7, 2017, the contents of which are herebyincorporated by reference.

FIELD OF INVENTION

The present invention is generally directed to the field of ophthalmicimaging systems. More specifically, it is directed to techniques formontaging two or more scans obtained with the ophthalmic imaging system.

BACKGROUND

A wide variety of interferometric based imaging techniques have beendeveloped to provide high resolution structural information of samplesin a range of applications. Optical Coherence Tomography (OCT) is aninterferometric technique that can provide images of samples includingtissue structure on the micron scale in situ and in real time (Huang, D.et al., Science 254, 1178-81, 1991). OCT is based on the principle oflow coherence interferometry (LCI) and determines the scattering profileof a sample along the OCT beam by detecting the interference of lightreflected from a sample and a reference beam (Fercher, A. F. et al.,Opt. Lett. 13, 186, 1988). Each scattering profile in the depthdirection (z) is reconstructed individually into an axial scan, orA-scan. Cross-sectional images (B-scans), and by extension 3D volumes,are built up from many A-scans, with the OCT beam moved to a set oftransverse (x and y) locations on the sample.

Optical coherence tomography (OCT) is a noninvasive, noncontact imagingmodality that uses coherence gating to obtain high-resolutioncross-sectional images of tissue microstructure. Several implementationsof OCT have been developed. In time domain OCT (TD-OCT), the path lengthbetween light returning from the sample and reference light istranslated longitudinally in time to recover the depth information inthe sample. In Frequency domain OCT (FD-OCT), the interferometric signalbetween light from a reference and the back-scattered light from asample point is recorded in the frequency domain either by using adispersive spectrometer in the detection arm in the case ofspectral-domain OCT (SD-OCT) or rapidly tuning a swept laser source inthe case of swept-source OCT (SS-OCT). After a wavelength calibration, aone-dimensional Fourier transform is taken to obtain an A-line spatialdistribution of the object scattering potential, e.g. an A-scan.

Functional OCT can provide important clinical information that is notavailable in typical OCT images, which are intensity based and providestructural information. There have been several functional contrastenhancement methods including Doppler OCT, Phase-sensitive OCTmeasurements, Polarization Sensitive OCT, Spectroscopic OCT, etc.Integration of functional extensions can greatly enhance thecapabilities of OCT for a range of applications in medicine.

One of the most promising functional extensions of OCT has been thefield of OCT angiography which is based on flow contrast. Visualizationof the detailed vasculature using OCT could enable doctors to obtain newand useful clinical information for diagnosis and management of eyediseases in a non-invasive manner. Fluorescein angiography andindocyanine green (ICG) angiography are currently the gold standards forvasculature visualization in the eye. However, the invasiveness of theseapproaches combined with possible complications (allergy to dyes andside effects) make them unsuitable techniques for widespread screeningapplications in ophthalmic clinics. There are several flow contrasttechniques in OCT imaging that utilize the change in data betweensuccessive B-scans or frames (inter-frame change analysis) of the OCTintensity or phase-resolved OCT data. A B-scan is a collection ofadjacent A-scan (typically arranged linearly) defining a two-dimensional(2D) image along an axial direction of a scan beam. One of the majorapplications of such techniques has been to generate en face vasculatureimages of the retina. An en face image, or face image projection is a 2Dfrontal view of segmented tissue layer. For example, in the case ofstructural OCT, an en face image may typically be generated byprojecting the average reflectance signal intensity over depth onto a 2Dcanvas, which may be parallel to a plane of the retina. High resolutionen face visualization based on inter-frame change analysis requires highdensity of sampling points and hence the time required to finish suchscans can be up to an order of magnitude higher compared to regular cubescans used in commercial OCT systems. A cube scan (or data cube orvolume) is a collection of adjacent B-scans that define athree-dimensional (3D) scan of a volume.

The large acquisition times and huge data volumes make it challenging toobtain high resolution data over large fields of view (FOV). Acquisitionof multiple smaller data cubes of smaller FOV and montaging themtogether to generate images and analysis over larger FOV has beendescribed (see for example, Y. Li et al., “Automatic montage of SD-OCTdata sets,” Optics Express, 19, 26239-26248 (2011) and US Pat. App. Pub.2013/0176532, the contents of which are hereby incorporated byreference).

Montaging is also used to expand the field of view in fundus imagingsystems. Montaging of fundus images can aid clinicians by providing amore complete view of the retina. Fundus image montaging is a commontechnique for extending the imaged field-of-view, and has been offeredas a feature on fundus cameras, including the Zeiss VISUCAM® andHeidelberg SPECTRALIS®. Applicants have described different aspects ofmontaging related to OCT and fundus imaging in the past (see for exampleUS Pat. App. Pub. No. 20170316565, International Pat. App.PCT/EP2018/071744, US Pat. App. Pub. 2013/0176532, and US Pat. App. Pub.No. 2016/0227999, the contents of which are hereby incorporated byreference).

It is an object of the present invention to provide improvements andenhancements to systems and methods for montaging image data of the eyeof a patient.

It is a further object of the present invention to provide amethod/system for optimizing/customizing scan patterns to the physicalcharacteristics of a patient's eye.

SUMMARY

The above objects are met in a system and method for improved montagingof retinal images. In one embodiment, a particular acquisition work flowis presented that reduces the total time the patient must remain stillin front of an imaging instrument. In another embodiment, an ophthalmicimaging system offers the user different montaging modes where theamount of overlap between the constituent images in the montage can bevaried depending upon the curvature of the eye. In yet anotherembodiment, montage configurations comprising images of different sizesand resolutions are described. In one or more embodiments, a preliminaryscan (or “prescan”) can be used to identify the optimal overlap betweenconstituent images to be montaged and to optimize the scan sizes (e.g.,dimension of scanned areas) and scan locations for a particular eye. Ina further embodiment, a system or method is described for applyingartifact removal to all the constituent images in a montage. In anotherembodiment, the system or method performs a quality check on themontaged image to confirm that the acquired, constituent images areplaced in their correct relative location within the final montage.

In embodiments, a method/system/device is provided for collecting a setof images for montaging. The set of images may be collected using anophthalmic imaging system, such as an OCT system or a fundus imager. Auser interface provides a scan option that includes two or more scanscovering different transverse regions on the retina of a patient, withsome overlap between pairs of regions. Upon selection of the scanoption, capture of the two or more scans for the selected scan option isinitiated. The collected two or more scans are displayed for approval.The collected two or more scans can be displayed separately or as amontaged image. In response to user selection of any of the displayedscans by use of a user input device, the selected scans areautomatically retaken and the display is updated with the recapturedscans. If the two or more scans are displayed separately for approval(e.g., not montaged), then the displayed scans may be montaged after auser input indicating user approval.

Alternatively, if the two or more scans are displayed as a montagedimage, the user selection may be applied to at least one constituentscan within the montaged image. Additionally, the two or more scans mayhave preassigned displacement positions relative to each other withinthe montaged image. In this case, a processor may process the montagedimage to determine if the two or more scans are in their preassigneddisplacement positions relative to each other in the montaged image, andidentify any misplaced scan or display an error indicator based on thedetermination. Optionally, a scan displacement input may be provided inthe user interface, wherein the preassigned displacement positionsrelative to each other of the two or more scans may be adjusted by useof the scan displacement input. Additionally, in response to failing tocapture any of the two or more scans for the selected scan option, thefailed scans may be excluded from the montaged image. In place of afailed scan, a failure indicator may be displayed within the montagedimage at a location corresponding to where failed scans would have beenif it had not failed. A user may select the failure indicator, which maycause an automatic recapturing of the scan corresponding to the selectedfailure indictors and an updating of the montaged image with therecaptured scan. Optionally, the failure indicators may include any of atextual description of failure, a graphic representation of failure,and/or a highlighted border indicating an outline of the failed scanwithin the montaged image.

Optionally, any of the collected scans may be retaken prior to montagingbased on an approval input from the user. That is, montaging may notnecessarily be initiated immediately after collecting the scansassociated with the selected scan option. Rather, the system or anoperator is given an opportunity to review the individual scans todetermine if they are of sufficient (e.g., minimum) quality prior tomontaging. For example, the system may determine a numerical qualityfactor (or measure) for each scan (e.g., ranging from 1 to 10, with 10indicating top quality), and any scan whose determined quality factor isnot higher than a predefined, minimum quality factor (e.g., 6), may berescanned until a scan meeting the minimum quality factor is achieved.Similarly, the operator may visually inspect the displayed scans anddetermine if they are of sufficient quality for montaging. The operatormay select (e.g., designate by use of a user input) any collected scanfor rescanning. Irrespective, after satisfactory scans are collected,the constituent scans (e.g., images) may be montaged, and the montagedimage may be stored, displayed, or submitted to further analysis.

In embodiments, the two or more scans of the scan option may coverdifferently sized regions of the retina. That is, each scan may span adifferently sized area of the retina, such as one scan spanning a 3×3 mmarea and another spanning a 6×6 mm area. Additionally, each scan may beof different resolution. For example, if the fundus imaging system is anOCT system and each scan region has an equal number of B-scans, thenchanging the size of a scan region will change the resolution of thatscan region.

Furthermore, the montaged image may be comprised of different fractionsof each of the two or more scans of the scan option. That is the systemmay identify the higher quality images in a scan option, and use largeramounts of the higher quality scans to construct the montaged image. Forexample, if the system assigns a quality factor, or other qualitymeasure, to each of the captured two or more scans, then the capturedscans having higher quality factors may comprise larger fractions of themontaged image, and scans having lower quality measures may make upsmaller portions of the montaged image.

The system may incorporate additional quality checks. For example, thetwo or more scans of the scan option may have preassigned displacementpositions/locations relative to each other, and the montaged image maybe checked to ensure that each scan is at its preassigned position. Anymisplaced scan may be identified and/or an error may be issued. Forinstance, if the scan option is comprised of two scans, then the systemmay assign the first scan to a left-most position in the montaged image,and the second scan to a right-most position in the montaged image. Thelocation of each scan within the eye may be controlled by use offixation light. In another example where the scan option is comprised offive scans, one scan may be preassigned a center location within themontaged image, and the remaining four scans may be preassigned (e.g.,by scan sequence) to specific quadrants along the periphery of thecenter scan.

Optionally, the user interface may include a user input for removingartifacts, such as image artifacts from individual scans and/or from themontaged image.

The present invention is also embodied in a method/system/device forgenerating a montaged fundus image of an eye. The montaged image may begenerated using an ophthalmic imaging system, which may be an OCT systemor a fundus imager. A user interface provides multiple scan options,where each scan option includes (e.g., defines) two or more scanscovering different transverse regions on the retina of a patient withsome overlap portion between scans. Each of the different scan options,however, may define a different amount of overlap among its respectivetwo or more scans. For example, one scan option may define more tightlyoverlapping scans than another. Upon selection of a scan option fromamong the multiple scan options, capture of the two or more scansincluded the selected scan option (e.g., defined by the selected scanoption) is initiated. The captured two or more scans may then becombined into a single montaged image, which may be stored, displayed orsubmitted for further analysis.

Optionally, the user interface further may provide a separate scanstatus for each of two or more scans during the capture of the two ormore scans. For example, if the two or more scans are collected insequence, the user interface may provide an indicator specifying whatportion of the retina is current being scanned (e.g. indicate which ofthe two or more scan is currently being collected, and/or which scan hasalready been collected). The user interface may also provide a userinput for recapturing a selected one or more of the two or more scansduring the capture of the two or more scans. For example, if theoperator notes a situation that may affect the scan (e.g., an eye blinkor patient movement), the operator may indicate that a current scan(among the two or more scans) should be rescanned.

Each of the plurality of scan options may be separately optimized for adifferent eye curvature. For example, eyes with higher curvature maybenefit from a scan option that provides a larger overlap among its twoor more scans. Optionally, a prescan of the eye (e.g., prior tocollecting the two or more scans of a scan option) may be collected todetermine an optimal scan pattern for use during the capture of the twoor more scans based on the curvature of the eye. That is, the prescanmay be used to determine an optimal scan pattern. For example, theprescan may determine if a designated scan would be ill-affected by thecurvature of the eye prior to the designated scan being collected. Insome embodiments, one of the multiple of scan options may be selected,or recommended, based on the determined optimal scan pattern. Theprescan may be of lower resolution than the capture of the two or moredesignated scans.

An example of the use of this prescan would be if the ophthalmic imagingsystem were an optical coherence tomographic (OCT) system, and the twoor more scans of one of the multiple scan options included a centralscan and a periphery scan that is peripheral to the central scan. Inthis case, prior to the capture of the central scan and/or the peripheryscan, a survey B-scan may be applied as a prescan within a regioncorresponding to where the periphery scan is to be captured. If aportion of the survey B-scan is not fully resolved within the imagingdepth capability of the OCT system, then the designated scan region ofthe periphery scan may be shifted (or offset) to increase its overlapwith the central scan and thus move it to less curved region of theretina. Conversely, if the survey B-scan is fully resolved within theimaging depth capability of the OCT system, then the designated scanregion of the periphery scan may be shifted to decrease its overlap withthe central scan, which increases the total area covered by the twoscans. This would provide for a larger field of view for the finalmontaged image. In this way the scan pattern of each scan option may befurther adjusted dependent upon the curvature of the patient's eye(e.g., as determined by the prescan).

Various embodiments may also include additional error detection. Forexample, the two or more scans of an individual scan option may bedesignated preassigned positions relative to each other. In this case,the montaged image may be processed to determine if its two or moreconstituent scans are in their respective, preassigned positionsrelative to each other. Any misplaced scan may be identified, or anerror message may be issued based on the determination.

To further improve the montaging of the two or more scans of a scanoption, the amount contributed by each constituent scan to the finalmontaged image may be made dependent upon the quality of eachconstituent scan. For example, a quality measure may be determined foreach of the captured two or more scans prior to montaging, and capturedscans having higher quality measures may be selected to make up largerportions of the final montaged image.

Other objects and attainments together with a fuller understanding ofthe invention will become apparent and appreciated by referring to thefollowing description and claims taken in conjunction with theaccompanying drawings.

The embodiments disclosed herein are only examples, and the scope ofthis disclosure is not limited to them. Embodiments according to theinvention are disclosed in the attached claims directed to a method, astorage medium, a system, a device and/or a computer program product,wherein any feature mentioned in one claim category, e.g. method, can beclaimed in another claim category, e.g. system, as well. Thedependencies or references back in the attached claims are chosen forformal reasons only. However, any subject matter resulting from adeliberate reference back to any previous claims can be claimed as well,so that any combination of claims and the features thereof are disclosedand can be claimed regardless of the dependencies chosen in the attachedclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

In the drawings wherein like reference symbols/characters refer to likeparts:

FIG. 1 illustrates a generalized FD-OCT system used to collect 3-D imagedata of the eye suitable for use with the present invention.

FIG. 2 illustrates an example of a slit scanning ophthalmic systemsuitable for collecting fundus images for use with aspects of thepresent application.

FIG. 3a illustrates an example acquisition workflow for collectingimages for montaging using an ophthalmic imaging system, such as the OCTsystem of FIG. 1 or the fundus imaging system of FIG. 2.

FIG. 3b illustrates a montaged image comprised of two constituent scans(images) of different quality, where equal amounts of each constituentimage are stitched into the montaged image irrespective of theirrespective image quality.

FIG. 3c shows another example of montaging two images.

FIG. 3d illustrates a scan that incorporates more of a higher qualityscan than a lower quality scan.

FIG. 4a illustrates an interface for accommodating variability in eyecurvature while optimizing the field of view of a montage.

FIG. 4b and FIG. 4c illustrate montaged images that result frommontaging constituent images with different amounts of overlap. Themontage of FIG. 4b has a smaller overlap between its constituent images,as is suitable for an emmetropic eye, and provides a larger FOV. Themontage of FIG. 4c has a larger overlap between its constituent images,as is suitable for a myopic eye, and provides a smaller FOV.

FIG. 4d illustrates a case where a montaged image is created using 4successful scans out of a 5 scan option where one scan failed, and acase where the montage is recreated with all 5 successful scans.

FIG. 5a illustrates an eye having a stronger curvature of the retina ata Region A as compared to a Region B.

FIG. 5b illustrates a scan configuration (including prescans) to beapplied to a scan option consisting of five (cube) scans havingpredefined positions relative to each other, and which correspond to thefive scan configuration of FIG. 4 a.

FIG. 5c illustrates how the depth coordinates of A-scans that comprise asurvey B-scan (which may be part of a prescan) may be calculated todetermine if B-scans in this area of the retina can be fully resolvedwithin the imaging depth of the (OCT) instrument.

FIG. 5d illustrates a case where a scan region is offset (shifted) dueto part of a survey B-scan not being fully resolved within the imagingdepth of the instrument.

FIG. 6a is an example showing a wider field of view volume scan thatcollects all of the data over a flat area of retinal tissue with goodresolution.

FIG. 6b illustrates an example requiring a larger number of volumes withsmaller transverse field of views to fully capture the retinal tissuewith high resolution due to the retinal tissue having substantially morecurvature.

FIG. 6c shows, in two dimensions, the montaging of multiple retinasamplings (scans) with different heights and widths, as indicated byhighlighted rectangles.

FIG. 6d shows a montaged image where flow projection artifact removalhas not been enabled, in this case via selection of a selectable userinput.

FIG. 6e shows a case where flow projection artifact removal has beenenabled by mean of the selectable user input of FIG. 6 d.

FIG. 7a illustrates a workflow for checking the locations of individual,constituent scans within a montage.

FIG. 7b illustrates the identifying of the locations of constituentimages in a montaged image. For example, the locations may be identifiedby identifying a pixel number from each constituent image and locatingthat pixel in the composite, montaged image, where an “x” indicatesspecific pixel locations.

FIG. 7c illustrates a montaged image corresponding to the mapping ofconstituent images of FIG. 7 b.

FIG. 8 illustrates a computer system suitable for use in variousembodiments of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

All patent and non-patent references cited within this specification areherein incorporated by reference in their entirety to the same extent asif the disclosure of each individual patent and non-patient referencewas specifically and individually indicated to be incorporated byreference in its entirely.

Example Optical Coherence Tomography (OCT) System

A generalized FD-OCT system used to collect 3-D image data of the eyesuitable for use with the present invention is illustrated in FIG. 1. AFD-OCT system 100 includes a light source, 101. Typical light sourcesinclude, but are not limited to, broadband light sources with shorttemporal coherence lengths or swept laser sources. A beam of light fromsource 101 is routed, typically by optical fiber 105, to illuminate thesample 110, a typical sample being tissues in the human eye. The source101 can be either a broadband light source with short temporal coherencelength in the case of SD-OCT or a wavelength tunable laser source in thecase of SS-OCT. The light is scanned, typically with a scanner 107between the output of the fiber and the sample, so that the beam oflight (dashed line 108) is scanned laterally (in x and y) over theregion of the sample to be imaged. Light scattered from the sample iscollected, typically into the same fiber 105 used to route the light forillumination. Reference light derived from the same source 101 travels aseparate path, in this case involving fiber 103 and retro-reflector 104with an adjustable optical delay. Those skilled in the art willrecognize that a transmissive reference path can also be used and thatthe adjustable delay could be placed in the sample or reference arm ofthe interferometer. Collected sample light is combined with referencelight, typically in a fiber coupler 102, to form light interference in adetector 120. Although a single fiber port is shown going to thedetector, those skilled in the art will recognize that various designsof interferometers can be used for balanced or unbalanced detection ofthe interference signal. The output from the detector 120 is supplied toa processor 121 that converts the observed interference into depthinformation of the sample. The results can be stored in the processor121 or other storage medium or displayed on display 122. The processingand storing functions may be localized within the OCT instrument orfunctions may be performed on an external processing unit (e.g.,computer system 1000 shown in FIG. 8) to which the collected data istransferred. This unit could be dedicated to data processing or performother tasks which are quite general and not dedicated to the OCT device.The processor 121 may contain for example a field-programmable gatearray (FPGA), a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a graphics processing unit (GPU), a system onchip (SoC) or a combination thereof, that performs some, or the entiredata processing steps, prior to passing on to the host processor or in aparallelized fashion. The display can be a traditional display or of thetouch screen type and can include a user interface for displayinginformation to and receiving information from an instrument operator, oruser. The user can interact with the display using any type of userinput as known to those skilled in the art including, but not limitedto, mouse, knobs, buttons, and touch screen.

The interference causes the intensity of the interfered light to varyacross the spectrum. The Fourier transform of the interference lightreveals the profile of scattering intensities at different path lengths,and therefore scattering as a function of depth (z-direction) in thesample. The profile of scattering as a function of depth is called anaxial scan (A-scan). A set of A-scans measured at neighboring locationsin the sample produces a cross-sectional image (tomogram or B-scan) ofthe sample. A collection of B-scans collected at different transverselocations on the sample makes up a data volume or cube. For a particularvolume of data, the term fast axis refers to the scan direction along asingle B-scan whereas slow axis refers to the axis along which multipleB-scans are collected. A variety of ways to create B-scans are known tothose skilled in the art including but not limited to along thehorizontal or x-direction, along the vertical or y-direction, along thediagonal of x and y, or in a circular or spiral pattern. A volume of 3Ddata can be processed to generate wide field fundus images (i.e., enface images) by assigning a single representative value for theintensity values (e.g. summation, integration, median value, minimumvalue, etc.) in all or a portion of the volume along an axis of thevolume (see for example U.S. Pat. Nos. 7,301,644 and 8,332,016, both ofwhich are hereby incorporated by reference in their entirety). Theseimages may be referred to as slab images.

The sample and reference arms in the interferometer could consist ofbulk-optics, fiber-optics or hybrid bulk-optic systems and could havedifferent architectures such as Michelson, Mach-Zehnder or common-pathbased designs as would be known to those skilled in the art. Light beamas used herein should be interpreted as any carefully directed lightpath. In time-domain systems, the reference arm needs to have a tunableoptical delay to generate interference. Balanced detection systems aretypically used in TD-OCT and SS-OCT systems, while spectrometers areused at the detection port for SD-OCT systems. The invention describedherein could be applied to any type of OCT system.

The above-described OCT employs a traditional point scanning, or flyingspot, technique where a single point of light is scanned across thesample, typically in two dimensions. It is to be understood, however,that the OCT system may be modified to employ any number of otherscanning techniques, including parallel techniques. In paralleltechniques, a series of spots (multi-beam), a line of light(line-field), or a two-dimensional field of light (partial-field andfull-field) is directed to the sample. The resulting reflected light iscombined with reference light and detected. Parallel techniques can beaccomplished in TD-OCT, SD-OCT or SS-OCT configurations. It is furtherto be understood that any scan manipulation (e.g., manipulation of cubescans, B-scans, and/or A-scans) described herein is compatible with anyOCT scanning technique. Several groups have reported on differentparallel FD-OCT configurations (Hiratsuka, H. et al., Opt. Lett. 23,1420, 1998; Zuluaga, A. F. et al., Opt. Lett. 24, 519-521, 1999;Grajciar, B. et al., Opt. Express 13, 1131, 2005; Blazkiewicz, P. etal., Appl. Opt. 44, 7722, 2005; Povaay, B. et al., Opt. Express 14,7661, 2006; Nakamura, Y. et al., Opt. Express 15, 7103, 2007; Lee, S.-W.et al., IEEE J. Sel. Topics Quantum Electron. 14, 50-55, 2008; Mujat, M.et al., Optical Coherence Tomography and Coherence Domain OpticalMethods in Biomedicine XIII 7168, 71681E, 2009; Bonin, T. et al., Opt.Lett. 35, 3432-4, 2010; Wieser, W. et al., Opt. Express 18, 14685-704,2010; Potsaid, B. et al., Opt. Express 18, 20029-48, 2010; Klein, T. etal., Biomed. Opt. Express 4, 619-34, 2013; Nankivil, D. et al., Opt.Lett. 39, 3740-3, 2014).

Furthermore, the OCT system may use any one of a number of OCTAngiography processing algorithms on OCT data collected at the same orapproximately the same transverse locations on a sample at differenttimes to identify and/or visualize regions of motion or flow. A typicalOCT angiography data set contains multiple scans repeated at the sametransverse locations. Motion contrast algorithms can be applied to theintensity information derived from the image data (intensity-basedalgorithm), the phase information from the image data (phase-basedalgorithm), or the complex image data (complex-based algorithm). An enface vasculature image is an image displaying motion contrast signal inwhich the data dimension corresponding to depth is displayed as a singlerepresentative value, typically by summing or integrating all or anisolated portion of the data as described above.

The OCT system discussed herein may provide 2D (i.e. cross-sectional)images, en-face images, 3-D images, metrics related to a healthcondition, and the like. This system may be used with any other system.The OCT system may be used to analyze any sample.

Example Slit Scanning Fundus Imaging System

FIG. 2 illustrates one example of a slit scanning ophthalmic systemsuitable for collecting fundus images for use with aspects of thepresent application. As depicted, the system 200 includes one or morelight sources 201, preferably a multi-color LED system or a laser systemin which the etendue has been suitably adjusted. An adjustable slit 202is positioned in front of the light source 201 to determine theillumination line width. This could also be established by the sourceindependent of a slit or aperture. In the embodiment shown on FIG. 2,the slit 202 remains static during the imaging but can be adjusted todifferent widths to allow for different confocality levels and differentapplications either for a particular scan or during the scan for use insuppressing reflexes. An objective lens 203 forms a pupil of the slit.The objective lens 203 can be any one of state of the art lensesincluding but not limited to refractive, diffractive, reflective, orhybrid lenses/systems. The light passes through a pupil splitting mirror204 and is directed towards a scanner 205. It is desirable to bring thescanning plane and the pupil plane as near together as possible toreduce vignetting in the system. Optional optics 208 may be included tomanipulate the optical distance between the images of the twocomponents. The main task of the pupil splitter 204 is to combine andsplit the illumination and detection beams and to aid in the suppressionof system reflexes. The scanner 205 could be a rotating galvo scanner orother types of scanners (i.e. piezo or voice coil). Depending on whetherthe pupil splitting is done before or after the scanner 205, thescanning could be broken into two steps wherein one scanner is in theillumination path and a separate scanner is in the detection path.Specific pupil splitting arrangements are described in detail in USPatent Publication No. 2015/0131050, the contents of which are herebyincorporated by reference).

From the scanner, the light passes through one or more optics, in thiscase a scanning lens (SL) 206 and an ophthalmic or ocular lens (OL) 207,that allow for the pupil of the eye 209 to be imaged to an image pupilof the system. One possible configuration for these optics is a Keplertype telescope wherein the distance between the two lenses is selectedto create an approximately telecentric intermediate fundus image (4-fconfiguration). The ophthalmic lens 207 could be a single lens, anachromatic lens, or an arrangement of different lenses. All lenses couldbe refractive, diffractive, reflective or hybrid as known to one skilledin the art. The focal length(s) of the ophthalmic lens 207, scan lens206 and the size and/or form of the pupil splitting mirror 204 andscanning mirrors 205 could be different depending on the desired fieldof view (FOV), and so an arrangement in which multiple components can beswitched in and out of the beam path, for example by using a flip inoptic, a motorized wheel, or a detachable optical element, depending onthe field of view can be envisioned. Since the field of view changeresults in a different beam size on the pupil, the pupil splitting canalso be changed in conjunction with the change to the FOV. It ispossible to have a 45°-60° field of view as is typical for funduscameras. Higher fields of view (60°-120°) may be desired for acombination of the Broad-Line Fundus Imager (BLFI) with other imagingmodalities such as optical coherence tomography (OCT). The upper limitfor the field of view will be determined by the accessible workingdistance in combination with the physiological conditions around thehuman eye. Because a typical human retina has a FOV of 140° horizontaland 80°-100° vertical, it may be desirable to have an asymmetrical fieldof view for the highest possible FOV on the system.

The light passes through the pupil of the eye 209 and is directedtowards the retinal surface. The scanner 205 adjusts the location of thelight on the retina or fundus such that a range of transverse locationson the eye are illuminated. Reflected or scattered light (or emittedlight in the case of fluorescence imaging) is directed back along thesame path as the illumination. At the pupil splitting mirror 204, thereflected light is separated from the illumination light and directedtowards a camera 210. An objective lens 211 exists in the detection pathto image the fundus to the camera 210. As is the case for objective lens203, objective lens 211 could be any type of refractive, diffractive,reflective or hybrid lens as is known by one skilled in the art.Additional details of the scanning, in particular, ways to reduceartifacts in the image, are described in PCT Publication No.WO2016/124644, the contents of which are hereby incorporated byreference.

The camera 210 is connected to a processor 212 and a display 213. Theprocessing and displaying modules can be included with the system 200itself or on a dedicated processing and displaying unit, such as acomputer system wherein data is passed from the camera 210 to thecomputer system over a cable or network including wireless networks. Thedisplay and processor can be an all in one unit. The display can be atraditional display or of the touch screen type and can include a userinterface for displaying information to and receiving information froman instrument operator or user. The user can interact with the displayusing any type of user input as known to those skilled in the artincluding, but not limited to, mouse, knobs, buttons, and touch screen.

It is desirable for the patient's gaze to remain fixed while imaging iscarried out. One way to achieve this is to provide a fixation targetthat the patient can be directed to stare at. Fixation targets can beinternal or external to instrument depending on what area of the eye isto be imaged. One embodiment of an internal fixation target is shown inFIG. 2 for a slit scanning system. The same concept can be applied tothe OCT system in FIG. 1. In addition to the primary light source 201used for imaging, a second optional light source 217, such as one ormore LEDs, can be positioned such that a light pattern is imaged to theretina using lens 216. A scanning element 215 can be positioned suchthat it is located at the pupil plane of the system so that the patternon the retina can be moved depending on the desired fixation location.When montaging fundus images, it is necessary to change the rotation ofthe eye relative to the instrument and collect two or more images atdifferent rotations. This is most easily accomplished by adjusting thefixation location in the system.

In the configuration shown in FIG. 2, scattered light returning from theeye 209 (e.g., a collection beam) is “descanned” by scanner 205 on itsway to pupil splitting mirror 204. That is, scanner 205 scans theillumination beam from pupil splitting mirror 204 to define a scanningillumination beam across eye 209, but since it also receives returninglight from eye 209 at the same scanning position, scanner 205 has theeffect of descanning the returning light (e.g., cancelling the scanningaction) to define a non-scanning (e.g., steady or stationary) collectionbeam sent to pupil splitting mirror 204, which in turn folds thecollection beam toward the camera 210. Thus, the collection beam (fromall scan positions of the scanning illumination beam) is applied to thesame sensor region of the camera 210. A full-frame image may be built up(e.g., in processor 212) from a composite of the individually capturedcollection beams (resulting from the scanning illumination beam), suchas by montaging, or stitching. However, other scanning configuration arealso contemplated, including ones where the illumination beam is scannedacross the eye 209 and the collection beam is scanned across a 2D-areaphotosensor array of the camera. PCT Publication WO 2012/059236 and USPatent Publication No. 2015/0131050, hereby incorporated by reference,describe several embodiments of slit scanning ophthalmoscopes includingother designs where the light is swept across the camera, and alsodescanned imaging schemes. While the detailed description is focused onslit scanning ophthalmoscopes, many of the system elements describedherein, in particular the user interface elements, could haveapplicability to any type of ophthalmic imaging and diagnostic systems.

Montage Workflow for Minimized Chin Time

FIG. 3a illustrates one potential acquisition workflow for collectingthe images for a montaged fundus image using any type of ophthalmicimaging system, for example the OCT system illustrated in FIG. 1 or thefundus imaging system illustrated in FIG. 2, among others, that isdesigned to minimize the patient chin time (the time it takes to acquireall necessary scans for a montage). In this embodiment, the acquisitionworkflow may be started (301) by the system operator for example byselection of a particular scan option from a user interface thatprovides one or more scan options, or automatically by the software upondetection of an eye within its view. Subsequently, the system sets thescan location (302) by adjusting either the fixation location or scanoffset or both according to a pre-defined arrangement for a particularscan option. The arrangement may consist of at least two scans of thesame or different field of views covering different transverse locationson the retina but containing some overlapping portion. The scans mayspan similar or different size areas. At this point, the patient mayoptionally be more carefully aligned (or realigned) to the instrument,either manually by the operator or using an automatic alignment by thesystem and the scan capture may be performed (303) manually or by meansof an assisted software. The workflow may perform steps 302 and 303until all pre-defined scans (e.g., needed for creation of a montagedimage) are acquired (304). Once completed (e.g., step 304=Yes), theinstrument operator may be presented with all (or a select group of)scans (305) in one or more scan quality check screens where poor qualityscans may be identified either manually upon visual review by the useror automatically using a quality assessment algorithm, which maydetermine a numerical quality factor for each scan. Any unacceptable(e.g., poor quality scans) may be re-captured (step 305). This mayeffectively create a confidence interval for each constituent image tobe reviewed and designated for rescanning, if necessary. The workflowmay repeat steps 302, 303, and 304 on those pre-identified scans. Onceall presented scans are acceptable (step 305), the operator or softwaremay end the workflow (306), and the acquired scans can be montagedtogether using any of the many well-known feature matching and blendingapproaches (see for example Brown et al. “Recognising panoramas”, InProceedings of the 9th International Conference on Computer Vision,Volume 2, pages 1218-1225, Nice, October 2003; Brown et al. “AutomaticPanoramic Image Stitching using Invariant Features”, D. G. Int. J ComputVision (2007) 74: 59; Shi et al. “Good Features to Track”, Proceedingsof the 9th IEEE Conference on Computer Vision and Pattern Recognition,Seattle, Springer, June 1994 hereby incorporated by reference). For OCTdata, the first step would typically be to generate one or more en faceimages from each set of scan data (e.g., set of OCT scans that define acube scan, or volume), and registering and/or montaging the en faceimages to each other. The en face images could be of the entire volumeof data or particular slabs within the volume data, as is well known inthe art. Automatically creating a composite montage from overlappingimages typically involves several steps:

-   -   1) Detect features and calculate descriptors for them.    -   2) Match features among the images.    -   3) Calculate initial transformations (‘homographies’) between        pairs of images that map the matching features onto each other.    -   4) Calculate an optimum set of homographies that provide the        best matching among all of the images simultaneously (‘bundle        adjustment’).    -   5) Calculate the best locations for the transitions among the        constituent images in the final montage (‘seaming’).    -   6) Smoothly transition from each image to the others to reduce        obvious boundaries (‘blending’).        Prior to these steps, distortion correction could be implemented        which may involve using a projection removal algorithm as        discussed in more detail below.

In some embodiments, each scan may contribute an equal amount, or afixed region, to the montage. Alternatively, the determined numericalquality factor of each scan may be used to determine how much each scancontributes to the final montage. This may help improve the quality ofthe overall montaged image.

It is important to have good quality constituent scans (images) tocreate good montages. Poor quality scans (images) may result in pooroverall montages that may be of limited use to a clinician/operator.Having a confidence interval for imaging/scanning would be beneficialfor the operator (or technician) collecting the images and would providean opportunity to determine whether an image should be taken again. Amontaged image would also benefit from an algorithm, or process, thatincorporates more of the better image(s) into the montage. For example,images that have clearer-looking regions of interest, such the macula oroptic nerve, may be selected to contribute those clear-looking regionsinto the montaged image. In this way, the montage would have the bestfocused macula, optic nerve and other lesions of interest.

FIG. 3b illustrates a montage of two scans 311 and 313 of differentquality, where equal amounts of each image are stitched into the montageirrespective of their respective quality. In this approach, equalamounts of two, four, or more images contribute to a montage no matterhow poor the constituent images may be. The constituent image 311 on theleft shows choroidal detail whereas the constituent image 313 on theright does not show this detail and presents the optic nerve as veryblurry.

Another example of montaging two images with equal weighting is shown inFIG. 3c . In this case, a first scan 321 on the right has a blurry opticnerve 323, and a second scan 325 on the left has a less blurry opticnerve 327. Ideally, one should use the less blurry optic nerve 327 fromthe second scan 327 in the construction of a montage, but since in thepresent embodiment the final montage 329 uses equal amounts from eachscan 321 and 325, the final montage 329 shows good choroidal detail buta blurry optic nerve, which is contributed from image 321.

In a preferred embodiment, upon collecting an image (e.g., taking aphoto or finishing a cube scan of a predefined region), a quality factorwould be determined and displayed. For example, the quality factor maybe given a numerical value from 1 to 10. Various techniques forassigning a quality factor, or measure, to an image are known in theart, and its particular implementation is not critical to the presentinvention. An example of assigning a quality metric to an image isdescribed in U.S. Pat. No. 9,778,021, incorporated herein in itsentirety by reference. If the quality factor is less than a predefinedminimum (e.g., 6 out of 10), it would be advisable to re-do the image.For example, the display may provide an indicator identifying the badimage and suggesting that it be rescanned, or the algorithm mayautomatically rescan any image whose quality factor is less than aminimum, unless otherwise instructed by the operator. In this manner,poor quality montages may be avoided, or minimized. Additionally, thequality factor may be used to allocate contributions from each of theconstituent scans to the final montaged image. For example, thealgorithm may use more of the better quality constituent scan(s) (asdetermined by a higher value quality factor) to make the montaged image,rather than blindly stitching together the nasal and the temporalconstituent images irrespective of whether one is of poorer quality thanthe other.

FIG. 3d illustrates a scan 331 that incorporates more of a higherquality scan than a lower quality scan. As shown, the choroidal detailis good throughout the montage, and both the optic nerve and macula areclear.

Curvature Considerations for OCT Montaging

The imaging depth of most standard OCT instruments is typically limitedto 3 mm. Since the goal of montaging is to obtain a wide field of view,the montage workflow would ideally consist of acquiring a number of setsof OCT scan data (e.g., cube scans) placed as far as possible from eachother so as to cover the widest field of view supported by theinstrument while still having some overlap among the sets of OCT scandata. As the retinal curvature varies across the population, with highmyopic eyes having stronger retinal curvature compared to emmetropic andhyperopic eyes, it would be more challenging to acquire an OCT cube scanat the peripheral region of the retina of a high myopic eyes withoutvignetting the B-scans. The optimal placement of the constituent scansof a montage would be different for a high myopic as compared to ahyperopic eye. Therefore, it is desirable to have different montageconfigurations available in an instrument to accommodate the variabilityin the eye's curvature across the population such as to optimize thefield of view of the montage.

FIG. 4a illustrates an interface for accommodating variability in eyecurvature while optimizing the field of view of a montage. In thisembodiment, an OCT Angiography montage is desired. Three different FOVindicating icons 401 (each of which defines a different scan option) arepresented to the user where “+” signs are used to indicate the (e.g.,central) location of the multiple cube scans relative to each otherwithin each scan option. Closely placed “+” signs indicate more tightlypacked constituent scans (e.g., cube scans) having higher amounts ofoverlap between them. It is noted that the constituent cube scans withina scan option may be of different sizes and/or different resolutions. Inthe present example, the FOV indicating icon 401 on the far leftindicates the most densely spaced cube scans, such as may be used for ahighly myopic patient (e.g., to reduce vignetting of B-scans) to obtaina better quality, but smaller size, montaged image. The other two of thethree FOV indicating icons 401 indicate a more spread out spacing of thecube scans, such as for use with eyes with larger curvatures.

As stated above, the “+” signs of the three FOV indicating icons 401,each indicates a relative location of multiple (e.g., five) scans, e.g.,cube scans. Region 402 of the user interface 400 identifies each cubescan (e.g., scan icons 1 to 5, which may further indicate the sequencein which each cube scan is collected) and may illustrate the positioningof the cube scans relative to each other. For example, scan icon 1 maycorrespond to a central (cube) scan, and scan icons 2, 3, 4 and 5 mayeach correspond to a peripheral (cube) scan (e.g., peripheral to centralscan 1). Optionally, the relative position of any of the scans indicatedin interface 400 may be individually adjusted relative to each other.Individual scan icons may be automatically adjusted, as explained below,and/or may be manually adjusted by dragging the position of any of thescan icons 1 to 5 to a new relative position within region 402 by use ofa user input device, e.g., a mouse pointer. It may be desirable toadjust the relative position of any scan (beyond the default, relativepositions of any given scan option 401) if, for example, it is foundthat a particular scan is of lower than desired quality or partiallyfailed (e.g., due to excessive curvature in the eye within the region ofthe failed scan). In this case, the undesired scan may be replaced by ahigher quality scan by moving the location of the scan to a less curvedsection of the eye, such as closer to the central area of the eye.

Since the multiple cube scans may be acquired independently (e.g., insequence), region 402 can be used to provide the user with a statusupdate on the scan acquisition process. For example, each scan icon 1 to5 in region 402 may be highlighted as its corresponding cube scan isbeing acquired, or is completed, as illustrated by highlighted scan icon1. For instance, in one embodiment, the different scan icons 1 to 5 canbe demarcated with a check-mark, or a highlighted border, or some othervisual indicator as their corresponding cube scan is acquired and/orcompleted. In addition, if a user selects (or moves) any of scan icon 1to 5 that indicates an already acquired scan, the system may respond byretaking that icon's corresponding cube scan for the montage. This maybe the case, for example, if the operator is aware of some condition,such as an eye blink or movement of the patient that may lead to areduced quality image. The cube scan's corresponding icon can behighlighted (e.g., by a blinking border or a differently colored border)to indicate that the cube scan is currently being re-acquired.

FIG. 4b and FIG. 4c illustrate montaged images that result frommontaging constituent images with different amounts of overlap (such asindicated by scan options 401). FIG. 4b is a wide configuration (e.g.,smaller overlap between sets of OCT scans) shown for an emmetropic eye.FIG. 4c is a narrow configuration (e.g., larger overlap between sets ofOCT scans) shown for a myopic eye. The curvature of the eye can make itdifficult to obtain OCT montages as it may lead to vignetting ofB-scans, for example, because the variation of the optical path lengthof the scanning points across the retina may exceed the imaging depth ofthe system. This may be especially bad in the presence of myopia asillustrated on the right edge in FIG. 4c . To overcome this problem whenacquiring multiple peripheral scans intended to be montaged, it ispossible to adjust the location of the scans relative to the opticalaxis of the instrument.

As stated above, a user may choose to retake any scan used in a montagedimage. The user may choose to retake an image before or after themontaged image is created. For example, the user interface may provide apreview screen on which each collected scan (as identified by scan icons1 to 5) is displayed separately. At any time the user may select analready collected scan (e.g., either from this preview screen or fromregion 402) for rescanning. Optionally, the collected scans may not bemontaged until after a user input indicating user approval is submitted.Further optionally, the collected images/scans may be montaged and theuser may select (e.g., by means of any known user input device, such asa computer mouse, stylus, touch-sensitive screen, etc.) an individualscan within an already montaged image, and have the individuallyselected scan retaken. The montaged image may then be updated with thenewly retaken scan.

Further alternatively, if a particular scan fails (such as if a scanfrom a selected scan option 401 fails), the montaged image may still becreated using the scans that did not fail. FIG. 4d illustrates a casewhere a montaged image 431 is created using 4 successful scans of a 5scan option where one scan failed, and a case where a montaged image 433is recreated with all 5 successful scans. In the montaged image 431 withone or more missing, failed scans, a visual indicator may be provided inthe montaged image 431 indicating which scan failed and is missing fromthe montaged image. This visual indicator may include a textual message425 (e.g., a description of the type of failure and/or messageidentifying the failed scan), a graphic 423 (e.g., a graphicrepresentation of failure and/or of a failed scan), and/or a highlightedborder 421 indicating an outline of the failed scan within the montagedimage. The visual indicator may further identify the scan number (e.g.,scan icons 1 to 5) that failed and may be located at the relativelocation of where the failed scan would have been in the montaged image431 if it had not failed. The user may then select this visual indicatorin the montaged image by use of a user input device, such as pointer427, and have the system retake the failed scan. The montaged image isthen updated, as illustrated by montaged image 433 to include theretaken scan. Additionally, the user may adjust the relative location ofthe failed scan prior to retaking the failed scan. For example, thepointer 427 may be used to drag the location of outline 421 closertoward the center of montaged image 431 prior to rescanning the failedscan.

An example of a mechanism for adjusting the relative location of a scanis provided in FIGS. 5a to 5d . FIG. 5a illustrates an eye 501 having astronger (e.g., higher) curvature of the retina at a Region A ascompared to a Region B. Consequently, scan beams 502 adjusted for theretina surface in region B may not be well-suited for Region A, asillustrated by scan beams 503. While acquiring multiple cube scans, acube scan placed in region A can be offset toward the central part ofthe retina (where the retina is flatter) to avoid vignetting, e.g.,vignetting of the cube scan's B-scans.

FIG. 5b illustrates a scan configuration (including prescans) for a scanoption consisting of five (cube) scans C1, P2, P3, P4, and P5 havingpredefined positions relative to each other, and which in the presentexample correspond to the five scan configuration of FIG. 4a . FIG. 5bshows an y-z view of an eye 501 on the left-hand side, and a scanconfiguration along the back (x-y view) of the eye 501 b. The scanconfiguration of FIG. 5b includes one central (cube) scan C1 (shown indark solid line) surrounded by four peripheral (cube) scans P2 to P5(shown as dash lines or solid gray lines). Since the central scan C1 islocated at an area of the retina having relatively low curvature, it mayoptionally be taken directly. The four periphery scans P2 to P5,however, span areas of the retina having higher curvatures. Therefore, aperiphery scan may be preceded by a corresponding survey scan (orprescan) of the retinal area that is to be scanned by the peripheryscan.

For illustration purposes, an example configuration of a survey scan asapplied to periphery scan P4 (located at a position lower-right tocentral scan C1) is shown. It is to be understood that a similar surveyscan may be applied to any cube scan in the scan configuration along theback of the eye 501 b. A survey scan may consist of one or more surveyB-scans to determine if the periphery scan should be offset, such as toobtain a larger FOV for the montaged image or to avoid errors such asvignetting. That is, the survey scans may be used to determine optimalpositions for their corresponding cube scans or portions of cube scans,e.g., P2 to P5. A survey scan may be of lower resolution than itscorresponding periphery scan, and its survey B-scan(s) may be located ator near the edges (e.g. along the margins) of the retinal area/regiondefined by the periphery scan that is to be collected. A survey scan mayinclude one or more survey B-scans along a first dimension (e.g., thex-axis). For example, a survey scan may have a top horizontal surveyB-scan SH1, a middle horizontal survey B-scan SH2, and a bottomhorizontal B-scan SH3. Optionally, a survey scan may also include one ormore vertical survey B-scans traversing the horizontal survey B-scans.The present example shows two vertical survey B-scans SV1 and SV2. Theindividual survey B-scans may then be examined to determine if theircorresponding periphery scan should be offset.

For example, FIG. 5c shows survey B-scan SH1 of FIG. 5B, and illustrateshow to determine if it is fully resolved. The depth coordinates of theA-scans of survey B-scan SH1 may be calculated to determine ifindividual A-scans are within the imaging depth range of the instrument.In FIG. 5c , all of the A-scans comprising survey B-scan SH1 are withinthe imaging depth range (along the z-axis) of the instrument, and sosurvey B-scan SH1 may be deemed to be fully resolved. Optionally,additional survey B-scans corresponding to periphery scan P4 (see FIG.5b ) may be examined to determine if they can be fully resolved. If all(or a select, representative sub-set of) the survey B-scans for aparticular periphery scan are fully resolved, then it can be expectedthat the periphery scan corresponding to these survey B-cans would alsobe fully resolved, and no scan offset is necessary.

By contrast, FIG. 5d illustrates a case where it is determined that acube scan should be offset, or shifted, due to part of at least one ofits corresponding survey B-scan not being fully resolved within theimaging depth range of the instrument. FIG. 5d reproduces eye 501 ofFIG. 5a and further demarcates on the eye the imaging depth of theinstrument. For illustration purposes, imaging window 511 shows surveyB-scan SH3, which corresponds to the bottom edge of the periphery scanP4 in FIG. 5b . Survey B-scan SH3 is shown to not be fully resolvedwithin the imaging depth range of the instrument, e.g., its right-mostportion 513 is cut-off. That is, the A-scans that correspond to portion513 are outside the imaging depth range, which may cause vignetting. TheA-scans in region 513 may also introduce mirror artifacts 515, such asdue to the complex conjugate effect of the Fourier transforms used toresolve A-scans.

To avoid vignetting and other artifacts, the application can calculate,from the number of A-scans that are not resolved within the imagingwindow 511, a scan offset to shift a corresponding cube scan closer tothe macula (and/or the central scan) corresponding to a “flatter” regionof the retina. For example, portion 513 may identify a scan offset(e.g., along the x-axis) to permit the survey B-scan SH3 to be fullyresolved. This scan offset may then be applied to the correspondingperiphery scan P4, as a whole, or individually to the correspondingB-scan(s) within periphery scan P4, to avoid artifacts in the montagedimage. For example, imaging window 517 shows a B-scan of the peripheryscan P4 that has been offset by an amount determined from portion 513.Consequently, all A-scans within this B-scan are fully resolved andvignetting is avoided.

However, if the survey B-scans are not vignetted, such as survey B-scanSH1 in FIG. 5c , the application may optionally determine a scan offsetto shift the corresponding cube scan further to the periphery of theretina (e.g. reduce its overlap with the central scan withoutintroducing vignetting) corresponding to a “steeper” region of theretina to get the largest field of view possible for the final montagedimage. This offset may be determined by extrapolating the survey B-scanSH1 beyond its imaging window, or by repeatedly shifting and retakingthe survey B-scan SH1 (while maintaining an overlap with the area ofcentral scan C1) until vignetting is encountered. Alternatively, theknown refractive error or axial length of the eye could be used topredict the best montage scan pattern for a particular eye.

Optionally, the above-described use of survey scans (e.g., prescans) todetermine preferred, or optimal, scan regions for periphery cube scansmay be used to automatically select, or suggest, one of the scan options401 (see FIG. 4a ) whose configuration is close (or closest) to thedetermined preferred scan regions. It is to be further understood thatthe above-described use of survey scans may also be used to modify anyselected scan option 401 to move (or recommend the moving of) any of itsindividual, constituent scans to positions more closely aligned withtheir preferred scan regions as determined from their correspondingsurvey scans. Thus, the location of any individual scan (e.g., asidentified by scan icons 1 to 5 in FIG. 4a ), may be individuallyadjusted/altered after selection of a specific scan option 401 by meanof a user input device or automatically in accordance with itscorrespondingly determined optimal scan region.

In another aspect of the present application, the size (e.g. area) ofthe cube scans collected for a montage can be varied. Larger size (e.g.,larger field of view) cube scans could be used centrally where theretina is flatter and smaller field of view cube scans could surroundthis central cube scan over areas where the retinal curvature preventslarger field of view cube scans. It may also be helpful to change theresolution of individual cube scans in either the transverse or axialdirections. For instance, near the fovea, it is desirable to havedensely sampled scans, but in the periphery, it may be sufficient tosample less densely. Alternately, it may be useful to have deeper scanswith lower axial resolution for more steeply curved eyes, such as thosewith myopia.

Some retinal tissue may appear flat over a large region. FIG. 6a is anexample showing a case where a wider field of view volume scan cancollect all of the data over a flat area of retinal tissue with goodresolution. Other retinal tissue may have substantially more curvature.FIG. 6b illustrates a case that requires a larger number of volumes withsmaller transverse field of views to fully capture the retinal tissuewith high resolution due to the retinal tissue having substantially morecurvature. FIG. 6c shows, in two dimensions, the montaging of multipleretina samplings (scans) with different heights and widths of scan.Individual retinal samplings are indicated by highlighted rectangularregions. The same principle would apply in three dimensions.

Flow Projection Correction

OCT Angiography images are susceptible to flow projection artifacts andmany different techniques for removing these projection artifacts havebeen proposed (see for example, “Projection-resolved optical coherencetomographic angiography”, by Miao Zhang et al., Biomed Opt Express. 2016Mar. 1; 7(3): 816-828; and “Minimizing projection artifacts for accuratepresentation of choroidal neovascularization in OCT micro-angiography”,by Anqi Zhang et al., Biomed Opt Express. 2015 Oct. 1; 6(10):4130-4143). In some cases, it is desirable for an OCT imaging system toallow the user to select whether or not to apply such a correction tothe imaging data for instance through the use of a user selectable iconor button on the user interface of the system or external processing.When the user enables the flow projection removal, the processor couldfunction to apply the projection artifact removal algorithm first to thedeeper slabs of all of the constituent en face images. Theprojection-artifact-free constituent images are then montaged asdescribed above. The user can also not select this artifact removalfunction and in this case the en face images are not corrected for flowprojection. FIG. 6d shows a montaged image where flow projectionartifact removal has not been enabled, in this case via selection of aselectable button 609, while FIG. 6e shows a case where the feature wasenabled by means of the selectable button 609. The removal, or lack ofremoval, of artifacts in images may be displayed/visualized using adepth-encoded, color coding scheme. For example, projection artifacts ina deeper slab may be displayed/identified using the a blue color.Consequently, the montaged image of FIG. 6e , which has projectionartifacts removed, may appear “redder” than when the projectionartifacts are not removed due to the blue color component of theprojection artifacts in the deeper slab having been removed.

Distortion correction of wide field OCT images prior to montaging

The scan angle at the pupil plane is not a linear function of thescanning element in the ophthalmic system, typically a galvanometer.This can lead to optical distortion in the image (a change ofmagnification off the field). The OCT image can be corrected fordistortion by encoding the x and y galvo positions such as to compensatefor the known distortion of the optical system—This correction can bedone while acquiring the scan rather than by post-processing of the enface image. The A-scans are no longer evenly distributed along theB-scans but have varying spacing over the length of the B-scans. Thissolution avoids an extra step in post-processing.

Montage Location Check

FIG. 7a illustrates a workflow for checking the locations of individual,constituent scans within a montage. Once montaging of the images iscomplete, it may be desirable to check the location of the individualscans in the montage. In an aspect of this invention shown in FIG. 7a ,the constituent images are received from the acquisition workflow (701).The images are montaged (702) and the locations of the constituentimages within the montaged image are identified (703) for example byidentifying a pixel number from each image and locating that pixel inthe composite image as illustrated in FIG. 7b where an “x” indicatespecific pixel locations. These locations can then be compared (704)with the relationships between the pre-defined scan locations (705). Forexample, if it is known that the “x” corresponding to image 1 should beabove the “x” of image 4, to the right of the “x” of image 0 and to theleft of the “x” of image 3, as is shown in FIG. 7b , then it can beinferred that image 1 is in its correct, designated position relative toimage 4, image 0 and image 3. But if the “x” of image 1 were not in itsexpected position relative to the other constituent images, then itcould be inferred that image 1 is not in its designated (expected)position, and an error notification could be issued. This can be used todetect misplacement of any constituent image within the final montagedimage (FIG. 7c ) and to inform the instrument operator (706) if amisplacement is detected. The user could then review the montaged imageand determine if there is in fact a misplaced constituent image, and ifso, retake the one or more misplaced images.

Example Computer System

Unless otherwise indicated, the processing units 121 and 212 that havebeen discussed herein (e.g., in reference to FIGS. 1 and 2) may beimplemented with a computer system configured to perform the functionsthat have been described herein for this unit. For instance, theprocessing unit 221 can be implemented with the computer system 1000, asshown in FIG. 8. The computer system 1000 may include one or moreprocessors 1002, one or more memories 1004, a communication unit 1008,an optional display 1010, one or more input devices 1012, and a datastore 1014. The display 1010 is shown with dotted lines to indicate itis an optional component, which, in some instances, may not be a part ofthe computer system 1000. In some embodiments, the display 1010 is thedisplay 122 or 213 that has been discussed herein in reference to FIGS.1 and 2.

The components 1002, 1004, 1008, 1010, 1012, and 1014 arecommunicatively coupled via a communication or system bus 1016. The bus1016 can include a conventional communication bus for transferring databetween components of a computing device or between computing devices.It should be understood that the computing system 1000 described hereinis not limited to these components and may include various operatingsystems, sensors, video processing components, input/output ports, userinterface devices (e.g., keyboards, pointing devices, displays,microphones, sound reproduction systems, and/or touch screens),additional processors, and other physical configurations.

The processor(s) 1002 may execute various hardware and/or softwarelogic, such as software instructions, by performing variousinput/output, logical, and/or mathematical operations. The processor(s)1002 may have various computing architectures to process data signalsincluding, for example, a complex instruction set computer (CISC)architecture, a reduced instruction set computer (RISC) architecture,and/or architecture implementing a combination of instruction sets. Theprocessor(s) 1002 may be physical and/or virtual, and may include asingle core or plurality of processing units and/or cores. In someembodiments, the processor(s) 1002 may be capable of generating andproviding electronic display signals to a display device, such as thedisplay 1010, supporting the display of images, capturing andtransmitting images, performing complex tasks including various types offeature extraction and sampling, etc. In some embodiments, theprocessor(s) 1002 may be coupled to the memory(ies) 1004 via adata/communication bus to access data and instructions therefrom andstore data therein. The bus 616 may couple the processor(s) 1002 to theother components of the computer system 1000, for example, thememory(ies) 1004, the communication unit 1008, or the data store 1014.

The memory(ies) 1004 may store instructions and/or data that may beexecuted by the processor(s) 1002. In some embodiments, the memory(ies)1004 may also be capable of storing other instructions and dataincluding, for example, an operating system, hardware drivers, othersoftware applications, databases, etc. The memory(ies) 1004 are coupledto the bus 1016 for communication with the processor(s) 602 and othercomponents of the computer system 1000. The memory(ies) 1004 may includea non-transitory computer-usable (e.g., readable, writeable, etc.)medium, which can be any apparatus or device that can contain, store,communicate, propagate or transport instructions, data, computerprograms, software, code, routines, etc. for processing by or inconnection with the processor(s) 1002. A non-transitory computer-usablestorage medium may include any and/or all computer-usable storage media.In some embodiments, the memory(ies) 1004 may include volatile memory,non-volatile memory, or both. For example, the memory(ies) 1004 mayinclude a dynamic random access memory (DRAM) device, a static randomaccess memory (SRAM) device, flash memory, a hard disk drive, a floppydisk drive, a CD ROM device, a DVD ROM device, a DVD RAM device, a DVDRW device, a flash memory device, or any other mass storage device knownfor storing instructions on a more permanent basis.

The computer system for the processing unit 121 or 212 may include oneor more computers or processing units at the same or differentlocations. When at different locations, the computers may be configuredto communicate with one another through a wired and/or wireless networkcommunication system, such as the communication unit 1008. Thecommunication unit 1008 may include network interface devices (I/F) forwired and wireless connectivity. For example, the communication unit1008 may include a CAT-type interface, USB interface, or SD interface,transceivers for sending and receiving signals using Wi-Fi™; Bluetooth®,or cellular communications for wireless communication, etc. Thecommunication unit 1008 can link the processor(s) 1002 to a computernetwork that may in turn be coupled to other processing systems.

The display 1010 represents any device equipped to display electronicimages and data as described herein. The display 1010 may be any of aconventional display device, monitor or screen, such as an organiclight-emitting diode (OLED) display, a liquid crystal display (LCD). Insome embodiments, the display 1010 is a touch-screen display capable ofreceiving input from one or more fingers of a user. For example, thedevice 1010 may be a capacitive touch-screen display capable ofdetecting and interpreting multiple points of contact with the displaysurface.

The input device(s) 1012 are any devices for inputting data on thecomputer system 1000. In some embodiments, an input device is atouch-screen display capable of receiving input from one or more fingersof the user. The functionality of the input device(s) 1012 and thedisplay 1010 may be integrated, and a user of the computer system 1000may interact with the system by contacting a surface of the display 1010using one or more fingers. In other embodiments, an input device is aseparate peripheral device or combination of devices. For example, theinput device(s) 1012 may include a keyboard (e.g., a QWERTY keyboard)and a pointing device (e.g., a mouse or touchpad). The input device(s)1012 may also include a microphone, a web camera, or other similar audioor video capture devices.

The data store 1014 can be an information source capable of storing andproviding access to data. In the depicted embodiment, the data store1014 is coupled for communication with the components 1002, 1004, 1008,1010, and 1012 of the computer system 1000 via the bus 1016.

In the above description, for purposes of explanation, numerous specificdetails are set forth in order to provide a thorough understanding ofthe specification. It should be apparent, however, that the subjectmatter of the present application can be practiced without thesespecific details. It should be understood that the reference in thespecification to “one embodiment”, “some embodiments”, or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin one or more embodiments of the description. The appearances of thephrase “in one embodiment” or “in some embodiments” in various places inthe specification are not necessarily all referring to the sameembodiment(s).

Furthermore, the description can take the form of a computer programproduct accessible from a computer-usable or computer-readable mediumproviding program code for use by or in connection with a computer orany instruction execution system. For the purposes of this description,a computer-usable or computer readable medium can be any apparatus thatcan contain, store, communicate, propagate, or transport the program foruse by or in connection with the instruction execution system,apparatus, or device. The foregoing description of the embodiments ofthe present subject matter has been presented for the purposes ofillustration and description. It is not intended to be exhaustive or tolimit the present embodiment of subject matter to the precise formdisclosed. Many modifications and variations are possible in light ofthe above teaching. It is intended that the scope of the presentembodiment of subject matter be limited not by this detaileddescription, but rather by the claims of this application. As will beunderstood by those familiar with the art, the present subject mattermay be embodied in other specific forms without departing from thespirit or essential characteristics thereof. Furthermore, it should beunderstood that the modules, routines, features, attributes,methodologies and other aspects of the present subject matter can beimplemented using hardware, firmware, software, or any combination ofthe three.

We claim:
 1. A method for collecting a set of images for montaging usingan ophthalmic imaging system, said method comprising: providing a scanoption via a user interface, wherein the scan option comprises two ormore scans covering different transverse regions on the retina of apatient, said regions containing some overlapping portion; uponselection of the scan option, initiating capture of the two or morescans for the selected scan option; and displaying the collected two ormore scans on a display for approval; in response to user selection ofany of the displayed scans by use of a user input device, automaticallyrecapturing the selected scans and updating the display with therecaptured scans.
 2. The method of claim 1, wherein the two or morescans displayed for approval are not montaged, and the displayed scansare montaged after a user input indicating user approval.
 3. The methodof claim 1, wherein the two or more scans displayed for approval aredisplayed as a montaged image, and said user selection is applied to atleast one constituent scan within the montaged image.
 4. The method asrecited in claim 3, wherein the two or more scans have preassigneddisplacement positions relative to each other within the montaged image,the method further comprising: processing the montaged image using aprocessor to determine if the two or more scans are in their preassigneddisplacement positions relative to each other in the montaged image; andidentifying any misplaced scan or displaying an error indicator based onthe determination.
 5. The method as recited in claim 3, wherein the twoor more scans have preassigned displacement positions relative to eachother within the montaged image, the method further comprising:providing a scan displacement input in the user interface, wherein thepreassigned displacement positions relative to each other of the two ormore scans is adjustable by use of the scan displacement input.
 6. Themethod of claim 3, further comprising: in response to failing to captureany of the two or more scans for the selected scan option, excluding thefailed scans from the montaged image, and displaying a failure indicatorwithin the montaged image for each failed scan at a location within themontaged image corresponding to where failed scans would have been if ithad not failed.
 7. The method of claim 6, further comprising: inresponse to user selection of any displayed failure indicator,automatically recapturing the scans corresponding to the selectedfailure indictors and updating the montaged image with the recapturedscan.
 8. The method of claim 7, wherein the failure indicators includeat least one of a textual description of failure, a graphicrepresentation of failure, and a highlighted border indicating anoutline of the failed scan within the montaged image.
 9. The method asrecited in claim 1, wherein the ophthalmic imaging system is one of afundus imaging system and an optical coherence tomography system. 10.The method as recited in claim 1, wherein the two or more scans coverdifferent sized regions of the retina.
 11. The method as recited inclaim 1, wherein the two or more scans have different scan resolutions.12. The method as recited in claim 1, further comprising: assigning aquality measure to each of the captured two or more scans; whereincaptured scans having higher quality measures comprise larger fractionsof the montaged image.
 13. The method as recited in claim 1, wherein theuser interface further includes a user input for removing artifacts. 14.A method for generating a montaged fundus image of an eye using anophthalmic imaging system, said method comprising: providing a pluralityof scan options via a user interface, wherein each scan option containstwo or more scans covering different transverse regions on the retina ofa patient, said regions containing some overlapping portion, wherein thescan options differ in the amount of overlap between their respectivelycontained two or more scans; upon selection of a scan option within theplurality of scan options, initiating capture of the two or more scansof the selected scan option; combining the two or more captured scansinto a single montaged image; and storing or displaying the montagedimage or a further analysis thereof.
 15. The method as recited in claim14, wherein the user interface further provides a separate scan statusfor each of two or more scans during the capture of the two or morescans.
 16. The method as recited in claim 15, wherein the user interfacefurther provides a user input for recapturing a selected one or more ofthe two or more scans during the capture of the two or more scans. 17.The method as recited in claim 14, wherein the ophthalmic imaging systemis an optical coherence tomographic system and each of the plurality ofscan options is separately optimized for a different eye curvature. 18.The method as recited in claim 14, further comprising performing aprescan of the eye to determine an optimal scan pattern for use duringthe capture of the two or more scans based on the curvature of the eye.19. The method as recited in claim 18, wherein the prescan is of lowerresolution than the capture of the two or more scans, and one of theplurality of scan options is selected based on the determined optimalscan pattern.
 20. The method as recited in claim 14, wherein theophthalmic imaging system is an optical coherence tomographic (OCT)system, and the two or more scans of one of the plurality of scanoptions includes a central scan and a periphery scan that is peripheralto the central scan, the method further comprising: prior to the captureof the periphery scan, collecting a survey scan within a regioncorresponding to where the periphery scan is to be captured; in responseto a portion of the survey scan not being fully resolved within theimaging depth capability of the OCT system, shifting the regioncorresponding to where the periphery scan is to be captured to increaseits overlap with the central scan.
 21. The method as recited in claim14, wherein the ophthalmic imaging system is an optical coherencetomographic (OCT) system, and the two or more scans of one of theplurality of scan options includes a central scan and a periphery scanthat is peripheral to the central scan, the method further comprising:prior to the capture of the periphery scan, collecting a survey scanwithin a region corresponding to where the periphery scan is to becaptured; in response to the survey scan being fully resolved within theimaging depth capability of the OCT system, doing one of the following:collecting the periphery scan within the region where the survey scanwas collected; or shifting the region corresponding to where theperiphery scan is to be captured to decrease its overlap with thecentral scan.
 22. The method as recited in claim 14, wherein the userinterface further includes a user input for removing artifacts from thetwo or more captured scans.
 23. The method as recited in claim 14,wherein the two or more scans have preassigned displacement positionsrelative to each other, the method further comprising: processing themontaged image using a processor to determine if the two or more scansare in their preassigned displacement positions relative to each otherin the montaged image; and identifying any misplaced scan or displayingan error indicator based on the determination.
 24. The method as recitedin claim 14, further comprising: assigning a quality measure to each ofthe captured two or more scans; wherein captured scans having higherquality measures comprise larger fractions of the montaged image.